The invention relates to a method and a system for visualizing the environment of a vehicle, as well as to a calibrating device for calibrating the system.
A night vision system for a vehicle is known from German Patent document DE 695 06 174 T2. The system has an infrared camera generating an infrared image, which shows the heat radiation emanating from persons and objects in a portion of the environment of the vehicle. The infrared image is projected by way of a head-up display on the lower portion of the windshield and is thereby made visible to the driver.
The assignment of persons and objects in the infrared image to the traffic situation situated in front of the driver and visually detectable through the windshield is to be made by the driver. Particularly in darkness, for which the known system is to be intended, this is difficult and often even impossible because the infrared image shows persons and objects which the driver cannot visually recognize. A more exact determination of the position of the persons and objects only recognizable in the infrared image is therefore not possible on a regular basis and dangerously diverts the driver's attention.
The object of the invention consists particularly of providing an improved method and an improved system for making the environment of a vehicle visible, such as particularly an improved night vision system.
The object of the invention is achieved with respect to the method by providing a visual image or its digital data of the environment, preferably a colored visual image, the visual image showing visible objects; and providing an infrared image or its digital data of the environment, the infrared image showing the infrared radiation emanating from the visible and/or other objects. And, a system is provided to implement the method. Other advantageous embodiments of the invention are described and claimed herein.
A first aspect of the invention consists of the display or visualization of a combination or superposition of a visual image, in the following called visual image, and of an infrared image of largely the same portion or cutout of the environment of a vehicle. As a result of this superposition or merging of the images of different spectral ranges of the same scene, the above-described disadvantages of the prior art are largely eliminated.
In a concrete further development of the night vision system according to the invention, the visual camera or the first sensor or their lens system has a first optical axis, and the infrared camera or the second optical sensor or their lens system has a second optical axis, which axes are spatially offset with respect to one another. The cameras or sensors therefore detect at least partially different cutouts of the environment of the vehicle, that is, a first and a second cutout. In addition, these are dependent on the distance. It is understood that, according to the invention, more than two infrared cameras or infrared sensors can be provided, whose sensitivity covers different wavelengths, and the digital images of the vehicle environment supplied by them can be superposed or merged.
In order to obtain a largely distortion-free merged image to be shown to the driver, it is provided according to a second aspect of the invention that the already provided first cutout and the already provided second cutout are completely or partially superposed or merged by a superposing or merging device with respect to pixels and/or areas. For providing the merged image, the first cutout and/or the second cutout and/or the merged image or their direct or processed digital data are adapted by at least one adaptation parameter.
This or these distance-dependent parameter(s) is/are preferably determined during the calibration of the camera system, sensor system, or night vision system for at least two distance ranges or spacing ranges between the cameras or sensors and a calibration device. It is a goal to adjust the adaptation parameter or parameters such that the merged image of objects, which forms during the superposition or merging of the images, is largely free of distortions in the respective distance range. In particularly, it should be free of ghost images or double images.
According to the invention, the adaptation parameter(s) is/are particularly at least one registration or transformation parameter. A similar parameter is known, for example, from the registration of a digital image processing system and superpositions of two images implemented thereby. The driving-situation-dependent or distance-dependent adaptation parameter(s) is/are preferably stored during the calibration in a data memory in the vehicle.
In a preferred embodiment of the invention, a first distance range corresponds to a driving situation typical of city driving, such as, in particular, a distance range of from approximately 15 to 75 m.
As an alternative or in addition, a second distance range corresponds to a driving situation typical of highway driving, such as, in particular, a distance range of from approximately 30 to 150 m.
As a further alternative or in addition, a third distance range corresponds to a driving situation typical of expressway driving, such as, in particular, a distance range of from approximately 50 to 250 m.
It is understood that, as an alternative or in addition to the above-mentioned distance-dependent or distance-range-dependent adaptation parameters, also other driving-situation-dependent adaptation parameters can be determined, particularly with the above-mentioned goal. These may, for example, be suitable adaptation parameters for driving while the sun is low, in fog, at dusk or in darkness.
A third aspect of the invention consists of automatically determining the actual driving situation of the vehicle and of providing the adaptation parameter(s) for use by an aligning device according to the invention, or for use by the superposing or merging device. The adaptation parameter(s) which correspond(s) to the actual driving situation, is/are obtained by calibration and is/are stored in the vehicle. The superposing or merging device carries out a pixel-type and/or area-type superposition or merging of the visual image, present in the form of digital data, and of the infrared image, in which case one or more distance-dependent adaptation parameter(s) influence(s) the infrared image and/or the visual image and/or the merged image such that, preferably, a largely distortion-free and/or ghost-image-free merged image of the environment is provided to the driver.
As an alternative or in addition to the above-mentioned automatic determination, in one embodiment of the invention, it is provided that the driver selects adaptation parameters which seem appropriate to him. The parameters are obtained particularly by calibration and are stored in the vehicle for use by the aligning device according to the invention or by the superposing and merging device. Thus, for example, the adaptation parameter or parameters for a driving situation typical of city driving, typical of highway driving, or typical of expressway driving, can be offered to the driver for a selection, for example, in the form of a manual selection possibility or by voice input. Thus, even without any vehicle navigation system situated in the vehicle, the driver is enabled to himself create a largely distortion-free or double-image-free merged image for the respective driving situation. In addition, the driver is offered the possibility of overcoming a possible automatic selection. Likewise, the driver can be provided with the possibility of selecting one or more adaptation parameters, which parameters display to him the close environment of his vehicle free of distortions in the merged image, for example, distances of up to 20 m from his vehicle. The driver can make this selection, for example, when he is approaching his garage in the dark and wants to find out by way of the infrared information in the merged image whether anyone is lying in wait for him behind a bush.
In a preferred embodiment of the invention, the actual position of the vehicle is determined by a vehicle navigation system, particularly a satellite navigation system. Using the position, the navigation system situated in the vehicle automatically determines, by use of the comparison with digital map data, the corresponding street or road category or driving situation, particularly a city street, highway or expressway. Such vehicle navigation systems already exist today in numerous vehicles for the purpose of routing, etc. and can be utilized without high expenditures for the above-mentioned automatic driving-situation-dependent optimization of the merged images of the environment of the vehicle.
As an alternative or in addition, it is provided in another embodiment of the invention that the driving situation is determined by use of at least one parameter of the driving dynamics, such as particularly the vehicle speed and/or the operation of the low beam or high beam and/or the vehicle acceleration and/or brightness and/or fog. This information can already be provided in today's vehicles completely or partially by its onboard power supply without requiring significant expenditures.
In a preferred embodiment of the invention, a colored visual image is merged, combined or superposed with the infrared image. In contrast to a black-and-white visual image, a merged image is created which contains the color information of the visual image, such as red brake lights of a vehicle driving ahead, a red traffic light, a colored traffic sign, or the like. The color information in the merged image facilitates the fast orientation and detection of the traffic situated illustrated in the merged image for the driver of a correspondingly equipped vehicle. In addition, the color information reduces the risk that colored warning signs (red traffic light, etc.) are overlooked.
Summarizing, the images of the environment or scene for different spectral ranges are in each case free of disturbing influences, such as distortions of the infrared or visual image, particularly because of imaging faults of the respectively used lens system, etc. With respect to the software, this preferably takes place by known measures for processing digitized images. The images or their digital image data, substantially free for the most part of disturbing influences, are preferably aligned with one another or caused to coincide with one another by digital image processing, so that largely isochronous and congruent image pairs consisting of the infrared and the visual image, or of their digital data, are present. According to the invention, this takes place by using at least one distance-dependent and/or driving-situation-dependent adaptation parameter for causing a distortion-free merged image. The software and hardware used for the digital image processing preferably permits a displacement, rotation, and scaling of the images relative to one another. As a result of this processing, the subsequent hardware expenditures for superposing or merging the images from different spectral ranges in a cost-effective manner—despite extensive real-time processing of isochronous and congruent image pairs—can be minimized.
According to a preferred embodiment of the invention, the infrared images and the visual images are generated with a respectively identical image repetition rate, preferably by one or two cameras or sensors for these spectral ranges. Isochronous image pairs can thereby be created in a particularly simple manner from the infrared image and the visual image, which considerably reduces the software and hardware expenditures for the subsequent superposing or merging of the two images according to the invention—largely in real time. The processing speed of the image pairs rises; and, the costs for semiconductor memory for the intermediate storage of the images are minimized.
In a preferred embodiment of the invention, the infrared image shows the infrared radiation or thermal radiation emitted by persons and objects, which is in the wavelength range of approximately 8 to 14 mm. An IR camera or an IR sensor is preferably used which is sensitive in a range of from approximately 8 to 10 mm. As a result, advantageously an infrared radiator or such an illumination (typical wavelength range of from approximately 800 nm to 2.5 mm) for the vehicle environment can be eliminated. A mutual blinding of oncoming vehicles, which in a known manner are each equipped with infrared lighting, does not occur. Likewise, the infrared image according to the invention is not limited to the range of an infrared lighting.
As an alternative or in addition, it is provided according to a fourth aspect of the invention that the method according to the invention and the system according to the invention, respectively, permit the showing of a merged image consisting of the visual image and the infrared image of the largely identical cutout of the environment of a vehicle in a display, which display has at least one merged area and at least one non-merged area, or an area not merged to the same extent or with the same weighting.
This fourth aspect according to the invention is based particularly on the following considerations and measures. The environment situated in front of the vehicle, as a rule, has numerous objects, the objects normally having different distances from the vehicle.
The following driving situation will now be considered as an example. A vehicle driving ahead on a highway at a close range of from 10 to 25 m has a shorter distance from the cameras of a following vehicle equipped according to the invention than an area of the right shoulder situated in the dark on which—in a not visible manner—pedestrians are situated farther toward the front. As mentioned above, the adaptation parameters suitable per se for the highway drive (typical distance range approximately 30 to 150 m) for the avoidance of ghost images or double images are used in the case of the method according to the invention and by the system according to the invention, respectively. Without any further influencing of the described merging of the visual image and of the infrared image, a merging image shown in the display is obtained whose center (or image center) shows the visual image of the vehicle driving ahead as well as an infrared image of the vehicle driving ahead which is at least partially spatially separate therefrom; that is, a ghost image or double image of the visual image is shown which irritates the driver. In contrast, the objects (for example, the pedestrians) situated within the range of from approximately 30 to 150 m are shown in the display largely without distortions or ghost images.
According to the invention, those areas of the merging image are not formed or merged or are formed or merged only to a reduced degree from the visual image and the infrared image which represent objects situated outside that distance range for which the actually used adaptation parameters are suitable. Thus, in the above-mentioned case, a merged image is created in the display which, in the center or image center, shows the visual image or the infrared image of the vehicle driving ahead (now without the double image) and around it, the merged image of the objects, such as the pedestrians, situated in the range of from 30 to 150 m, without ghost images.
In a first embodiment of the invention according to the fourth aspect, it is provided to measure the distance of the object driving ahead, for example, by means of radar, lidar (light detection and ranging) or ultrasound. According to a first variant, it is checked whether the determined spacing falls into the spacing range of the actually used adaptation parameters (for example, for highway driving). If so, no distortion is to be expected; that is, the merging can take place in the above-described manner. If not, the object or the concerned distance range, in a first variant, is shown in the display in the form of its visual image or its infrared image. If not, the object or the concerned distance range, in a second variant, is shown in the display by a partially merged image, the partially merged image being formed of the visual image and the infrared image of the object while using first adaptation parameters, and the remaining merged image or part thereof being formed by using second adaptation parameters. In the above-mentioned example, the first adaptation parameter or parameters is/are suitable for the distance of the vehicle driving ahead (approximately 10 to 25 m), and the second adaptation parameter or parameters is/are suitable for highway driving (approximately 30 to 150 m); that is, the result is again a distortion-free merged image (without areas not formed by merging) in the display.
It is understood that the change of the adaptation parameters used from one partially merged image to another partially merged image of the entire merged image illustrated in the display does not necessarily have to take place abruptly; to the contrary, it may be expedient to carry out the change in a flowing manner in order to obtain a largely distortion-free merged image of the objects consisting of different distance ranges (environment of the vehicle).
The use of radar, lidar or ultrasound for determining the distance is advantageous because corresponding transmitting and receiving devices are already available in modern vehicles for the ACC (automatic cruise control by radar or lidar) or for the purpose of parking in parking spaces (ultrasound).
It is understood that, as an alternative or in addition, it can be achieved by digital image processing that the adaptation parameter or parameters is/are varied in a suitable manner when merging the visual image and the infrared image for providing a distortion-free image of the environment of the vehicle (with objects in different distance ranges) in the display or the like.
The use of the above-mentioned transmitting and receiving devices for determining the distance, particularly of a vehicle driving ahead, should be advantageous in that the computing expenditures, and thus the hardware expenditures, for the digital image processing can be reduced in a cost-effective manner in the case of the method according to the invention and in the case of the system according to the invention, respectively.
Other objects, advantages and novel features of the present invention will become apparent from the following detailed description of the invention when considered in conjunction with the accompanying drawings.